Skip to content

AI Test Toolkit: language-first data-driven eval support#9460

Draft
thloke wants to merge 10 commits into
mainfrom
private/thloke/ait-language-first-datadriven
Draft

AI Test Toolkit: language-first data-driven eval support#9460
thloke wants to merge 10 commits into
mainfrom
private/thloke/ait-language-first-datadriven

Conversation

@thloke

@thloke thloke commented Jul 15, 2026

Copy link
Copy Markdown

Summary

Adds language-first ([TestDataSource]) data-driven support to the AI Test Toolkit so AIT evals can run on the platform test runner, coexisting with the classic app-based Eval Suite runner. Companion to platform (BC-Platform) and compiler (BC-DeveloperExperience) PRs.

Changes (all under src/Tools/AI Test Toolkit)

  • Shared, reusable building blocks (src/DataDriven/): AIT Test Case Context interface (extends ITestContext, exposes the classic AIT Test Context surface); AIT DD Test Context (implements it); AIT Test Data Source (implements ITestDataSource, resolves the dataset from the AIT run context or the attribute literal, one context per Test Input row); AIT Test Handler (implements ITestHandler) + enum registration; AIT DD Current Case (dataset-lineage bridge + session run id + per-case baseline).
  • Platform-runner logging: the handler writes one AIT Log Entry per case directly in OnAfterTestCaseRun (relies on the platform change that runs handlers outside function isolation). AddDataDrivenLogEntry + IsRunningUnderAITSuite gate so the classic suite path and the platform-runner path never double-log.
  • Suite coexistence / auto-detect: language-first codeunits are detected (CodeUnit Metadata."Has Test Data Source") and added once (no per-row expansion); Language First flag on the eval line as an explicit override.
  • Docs: README authoring + migration guide.

Migration surface for consumers

A typical eval migrates with three declarations and no new objects: [TestDataSource(Codeunit::"AIT Test Data Source", '<dataset>')], a context: interface "AIT Test Case Context" parameter, and TestHandlers = "AIT Test Handler".

Validation

Validated via al runtests against a local BC 29.0 server (provider + context fan-out, handler logging). Depends on the platform/compiler PRs to land for full behavior; the Context.Skip() credit-limit re-hook and streaming Harms provider are follow-ups once those ship.

Draft.

thloke and others added 5 commits July 15, 2026 10:48
…ce) support

Adds a shared ITestDataSource provider, an 'AIT Test Case Context' interface
(extends ITestContext) and a delegating context, a per-case dataset-lineage
bridge for logging, dual-mode dataset resolution, and coexistence with the
legacy per-row expansion (auto-detected via CodeUnit Metadata.HasTestDataSource,
with a Language First line flag as override). Docs updated.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n pattern

Adds a step-by-step migration recipe (attribute + context-parameter + drop the
AIT Test Context var), before/after example, rules (whole-codeunit, auto-detect),
variations (multi-turn, harms/adversarial, custom providers) and a checklist.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrating AIT evals to the language-first [TestDataSource] construct moves the
per-case fan-out to the platform test runner, so the toolkit's classic per-case
bracketing (Test Runner - Mgt.OnBefore/OnAfterTestMethodRun subscribers, which only
run under an AIT suite) no longer engages. This adds the platform-native seam:

- codeunit 149050 "AIT Test Handler" implements System.Testability.ITestHandler.
  OnBeforeTestCaseRun resets per-case accuracy/turns/token accounting and opens the
  run-procedure output scope; OnAfterTestCaseRun records the case outcome. Consuming
  eval codeunits opt in via TestHandlers = "AIT Test Handler".
- enumextension 149035 registers it under the platform TestHandler enum.
- "AIT DD Current Case" now buffers the per-case identity/outcome (in memory, so it
  survives the function isolation rollback) and flushes one AIT Log Entry from the
  out-of-isolation OnAfterTestMethodRun seam (decision C8-A). Note: that seam is
  raised by the Test Runner - Mgt runner (AL Test Tool / Test Suite Mgt); the pure
  platform test-execution path (al CLI) does not raise it, so log persistence there
  is a tracked follow-up.
- "AIT Test Suite Mgt.".AddDataDrivenLogEntry writes a per-case log entry with a
  session-scoped Run ID when there is no AIT suite; "AIT Test Run Iteration".
  IsRunningUnderAITSuite gates the handler off under an AIT suite to avoid double
  logging.
- Register the new objects in the toolkit object permission set and document the
  TestHandlers step in the migration guide.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AddDataDrivenLogEntry now calls AgentTestContextImpl.LogAgentTasks after inserting
the AIT Log Entry, matching the classic AddLogEntry path so agent-eval task
correlation is preserved for language-first evals running on the platform runner.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion)

Now that the platform runs ITestHandler hooks outside the per-function isolation
scope, the handler can write the AIT Log Entry directly in OnAfterTestCaseRun and it
persists past the case rollback. Remove the buffer/flush workaround: the pending-case
state and the OnAfterTestMethodRun flush subscriber in AIT DD Current Case, and
AIT DD Current Case is reduced to the dataset-lineage bridge + RunId + case-start
baseline.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Could not find a linked ADO work item. Please link one by using the pattern 'AB#' followed by the relevant work item number. You may use the 'Fixes' keyword to automatically resolve the work item when the pull request is merged. E.g. 'Fixes AB#1234'

thloke and others added 4 commits July 16, 2026 12:36
…osed dataset

Uptakes the reshaped platform ITestDataSource contract (enumerate + retrieve):
 - AIT Test Data Source: GetDataRows(List of [ITestContext]) -> ListTestCases(): List of [Text]
   (enumerate all dataset row codes up front) + GetTestCase(index, id): interface ITestContext
   (materialize one row's context on demand, only for cases that actually run).
 - AIT Test Case Context: declare Identifier() explicitly now that the base platform ITestContext
   is a pure marker, preserving the eval-facing API.
 - Fail-closed (D6): an unresolvable dataset id, or a dataset that resolves to zero rows, now errors
   instead of silently producing zero cases (which the runtime would treat as a false-green pass).

Compiles cleanly (76 files) against the reshaped System symbols + updated compiler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…only (D8)

Language-first detection is now solely CodeUnit Metadata.\"Has Test Data Source\" (platform field
12). Removes the interim \"Language First\" line flag (AIT Test Method Line field 8, its page control,
and the import/export XMLPort field-attribute) and the OR'd fallback in ExpandCodeunit. No backward
compatibility to preserve (unreleased).

Toolkit compiles cleanly against the reshaped symbols.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Addresses three single-instance state bugs that dropped or mis-attributed per-case logs when a
language-first eval runs on the platform test runner:
 - B2 (output lost): AddDataDrivenLogEntry read the per-case output from the handler's own
   (non-SingleInstance) "AIT Test Suite Mgt." instance, which is always empty. It now reads through
   the SingleInstance "AIT Test Context Impl." (new ConsumeRunProcedureOutput) - the same instance the
   test body wrote the output to.
 - B1 (stale lineage): the current case was bound lazily (only when the body first read its input), so
   a case that never touched its context logged the previous case's dataset lineage. GetTestCase now
   binds the current case eagerly at materialization (runs once per executing case, before the body and
   the OnAfterTestCaseRun log write).
 - B3 (leaked suite flag): AIT Test Run Iteration set ActiveAITTestSuite but never cleared it, so a
   standalone platform-runner data-driven run in the same session was mistaken for a suite run and the
   handler skipped its per-case logging. It is now cleared after each iteration.

Toolkit compiles cleanly against the reshaped symbols. NOTE: deferred (need the full toolkit app stack
to validate, or a design decision): the OnBeforeTestCaseRun scenario-start turn-count ordering, and the
language-first credit-limit skip re-hook (the limit provider is an AIT-suite concept; standalone
platform-runner runs have no suite).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ta-driven evals

Re-hooks the credit-limit skip for language-first evals run on the platform test runner (no AIT
suite). AIT Test Handler.OnBeforeTestCaseRun now checks the global monthly Copilot-credit limit
(AIT Eval Monthly Copilot Cred., an environment/company budget - not suite-scoped) and, when
enforcement is enabled and the limit is reached, calls Context.Skip() (reported Skipped by the
platform runner) and writes a Skipped AIT Log Entry via the new LogSkippedDataDrivenEval (mirrors
AddDataDrivenLogEntry with Status::Skipped + reason, keyed by RunId/codeunit/procedure/case; the
existing LogSkippedEval is suite-only). Enforcement disabled (default) -> IsLimitReached is false
-> no-op. Mirrors the classic app-suite behavior in AIT Test Run Iteration.OnBeforeTestMethodRun.

Toolkit compiles cleanly against the reshaped symbols.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@JesperSchulz JesperSchulz added the Integration GitHub request for Integration area label Jul 16, 2026
Extract shared helpers InitDataDrivenLogEntryHeader and EnrichLogEntryFromTestInput
so AddLogEntry, AddDataDrivenLogEntry and LogSkippedDataDrivenEval no longer duplicate
the header population and Test Input enrichment. Centralize the AutoIncrement
"Entry No." := 0 idiom with an explanatory comment. Also fix the inverted duration
guard in AddLogEntry (= 0DT -> <> 0DT) that the duplication had let drift.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: da7391b0-95a6-49b9-9a9a-14397106dd79
@thloke
thloke force-pushed the private/thloke/ait-language-first-datadriven branch from 2fb8a66 to 2b12d6c Compare July 17, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AL: Tools Integration GitHub request for Integration area

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants